ABSTRACT
Objectives: Use of real-world data/real-world evidence (RWD/RWE) in the life sciences is accelerating. The FDA has issued draft guidance for the conduct of real-world data-driven studies in clinical development. However, RWD protocol development standards lag, leading to heterogeneity of findings and consequent unreliability of results. The need to address challenges has become urgent due to increasing importance of reliable evidence generation in the COVID-19 pandemic. We performed a systematic review of published RWD protocols to understand current practices to support improvement in standards frameworks. Methods: We extracted protocols referencing RWD from. We defined essential real-world study concepts and mapped them to standard discrete protocol components. We summarized these components, including but not limited to: objectives, operational definitions of endpoints, inclusion/exclusion criteria, patient identification algorithms, schematics, extract/transform/load (ETL) methods, common data model (CDM), safety, analysis, and machine learning (ML)/artificial intelligence (AI). We identified areas of harmonization and disagreement, as well as missing components. Results: The search identified 220 real-world protocols. Despite substantial harmonization in some areas, particularly those components typical to all research studies, there was considerable disagreement regarding the representation of RWD objectives, RWD inclusion/exclusion criteria, data management, ETL, CDM, ML/AI, study design, and statistical analysis. In many cases, studies did not include real-world-specific elements at all. Quantification and statistical attribution of heterogeneity is ongoing. Conclusions: Incorporating best practices and harmonization of protocol development methods and reporting may lead to improved quality, consistency, and reproducibility of studies. The primary limitation of this study was that “real-world” was neither sensitive nor specific as a search term as it is often used imprecisely. Follow-up surveillance studies will quantify and evaluate the impact of improved standards, encompassing all registered observational studies.
ABSTRACT
Background: One-quarter of SARS-COV-2 (aka COVID-19) confirmed cases and deaths globally are in the United States (US), underscoring the need for a rigorous predictive model to inform healthcare decision making. Various forecast models have been developed, but very few use machine learning which typically offers greater predictive accuracy than traditional approaches. Objective: To develop a proof-of-concept dynamic geospatial model for predicting positive new SARS-COV-2 cases in US states using the transparent Bayesian networks machine learning approach. Methods: Targeted literature reviews were used to identify important predictive variables for positive new SARS-COV-2 cases. State-level data specifying identified variables were pooled from public sources, and final variable selection was informed by principal component analyses. A Bayesian network machine learning approach was used to identify interdependencies between variables. The outcome of interest was the predicted number of new positive SARS-COV-2 cases in each US state and county in 28 days from an index date. The model was trained with data from 40 randomly selected states and validated by K-fold cross validation. Model goodness-of-fit was assessed using visual inspection, AIC and Bayesian information criterion (BIC). Measures of predictive accuracy included root mean square error (RMSE), mean percentage error (MPE), mean absolute percentage error (MAPE) and area under the receiver operating characteristic curve (AUC - using a discretized outcome). Results: Predictions from the dynamic Bayesian model were a close fit to observed data for all states. The ME, RMSE, MAE, MPE and MAPE were 61.5, 497.9, 320.4, 36.0 and 49.1 respectively. Conclusion: We developed a geospatial dynamic Bayesian network model that accurately predicts positive new SARS-COV-2 cases in 28 days into the future for US states. The model’s prediction accuracy appears to be at least on par with other popular models that are available for public use.